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Abstract 


The Elementary Support Model (ESM) was implemented in Wake County Public School System’s 
(WCPSS) 12 lowest performing elementary schools. ESM is designed to improve teacher and student 
outcomes by addressing governance, staffing, professional development, resources, calendar and 
schedule, and provides leadership and instructional coaching. Qualitative methods were used to 
assess implementation and a quasi-experimental design was utilized to examine ESM’s intermediate 
and long-term outcomes. The implementation of ESM in WCPSS was strong. ESM was also successful 


in meeting, or mostly meeting, its short-term goals related to professional development, coaching, 
and governance. Results related to intermediate and long-term goals were mixed. Three areas of 
strength were principal support to teachers, reduction of beginning teachers, and increases in the 
number of ESM schools meeting EVAAS growth; however, additional goals related to student 
outcomes were either only partially met or not met. Recommendations focus on strengthening 
implementation and continuing to monitor progress toward reaching program goals. 
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The Elementary Support Model (ESM), implemented in 2015-16, is a Wake County Public School System 
(WCPSS) initiative designed to improve teacher and student outcomes for the district’s lowest 
performing schools. Twelve elementary schools’ comprise the WCPSS’ ESM Area and are supported by 
an area superintendent along with a cross-functional instructional team. Twelve elementary schools 
were selected (based on a multiple indicators ranking) as ESM schools to receive additional supports 
beginning in 2015-16. The model addresses governance, staffing, professional development, resources, 
calendar and schedule, and provides leadership and instructional coaching. 


Results 


As of 2016-17 (the second year of implementation), the implementation of ESM in WCPSS was strong: 
four implementation goals related to governance, instructional coaching, and staffing were fully met; 
and two goals related to classroom walkthroughs or “look fors” and the adoption of calendars and 
schedules were mostly met. ESM was also successful in meeting or mostly meeting its short-term goals 
related to professional development, coaching, and governance. Only one short-term goal—teachers 
implementing targeted best practices with fidelity—could not be determined as “Met” due to lack of 
classroom observation data. 


Results related to intermediate and long-term ESM goals as of 2017-18 were mixed. Goals related to 
principal support to teachers, reduction of beginning teachers, and increases in the number of ESM 
schools meeting Education Value-Added Assessment System (EVAAS) growth were met. Goals related to 
teacher absenteeism, teacher evaluation ratings, student absenteeism, teacher-student relationships, 
achievement gaps, and sixth grade achievement were only partially met, and goals related to proficiency 
goals related to Dynamic Indicators of Basic Early Literacy Skills® (DIBELS) composite, End-of-Grade 
(EOG) proficiency, and exceeding EVAAS growth were not met. 


Recommendations 


The following recommendations are based on the findings in this study: 


e More consistent school to school data collection efforts. Clarify expectations regarding the 
completion of student surveys. Centralize the collection of walkthrough or “look for” data. 

e Utilize an observational tool to inform instruction. School staff should use a ‘look for’ tool in order 
to inform teacher planning and monitor the success of classroom implementation of professional 
learning and the application of learning shared via coaching support. Use of the ‘Look-for’ should 
provide school administrators and other school staff formative information on classroom practices 
going forward. 

e Unpack principal concerns regarding ESM Coaches’ roles. Based on principal feedback, the role of 
the two full-time ESM coaches should be more flexible and responsive to school needs. ESM support 
teams focus group feedback indicated that beginning in 2017-18, each individual school’s needs 
would guide the coach’s role at that school. 


1 Barwell Road, Brentwood, Bugg, Creech Road, East Garner, Fox Road, Hodge Road, Lincoln Heights, Lynn Road, Smith, 
Walnut Creek, Wilburn. 
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e Clarification regarding accessing support from coordinating teachers. ESM principals requested 
improved communication for identifying (and the process for receiving) support from coordinating 
teachers needs clarification. 

e Continue to align ESM goals with MTSS Framework. Given that 2017-18 represents the third year of 
ESM implementation, and according to implementation science it takes three to five years to realize 
full implementation, ESM staff should revise (annually) and continue use of the ESM logic model as a 
tool for tracking progress toward reaching ESM goals. Furthermore, since the MTSS Framework is 
the broader umbrella under which the ESM initiative fits, staff should ensure ESM goals continue to 
align with the MTSS Framework. 

e Use successful ESM strategies and ESM school exemplars to inform the district’s Collaborative 
System of School Support. ESM’s structured support and leadership/mentoring strategy have 
shown success in reducing the number of beginning teachers, increasing the percentage of teachers 
feeling consistently supported by leadership, and increasing school’s meeting growth. 


Background 


In 2015-16, WCPSS began providing support to 12 elementary schools grouped within the newly created 
ESM Area based on multiple indicators including achievement, teacher characteristics, demographics, 
school climate, and school leadership. The 12 elementary schools rated lowest on these indicators were 
grouped for supervision by a newly hired Area Superintendent. This area superintendent was tasked 
with the creation of the ESM framework and strategies and was supported by a cross-functional 
instructional team. The ESM model that was developed addresses governance, staffing, professional 
development, resources, calendar and schedule and provides instructional coaching. Due to a change of 
leadership midway through ESM’s first year of implementation when the first ESM Area Superintendent 
left the district, a new ESM Area Superintendent had to be selected, which resulted in some changes in 
terms of how schools were supported. While all ESM schools continued to be required to participate in 
structured meetings, beginning in 2016-17 ESM central staff also tiered additional support (e.g., more 
frequent site visits) to schools based on need (e.g., school performance data and/or school leadership 
changes). While ESM’s basic structure remained the same (with some modifications), the new ESM Area 
Superintendent added an instructional focus, described as “back to basics.” In 2015-16, the structural 
support was established (i.e., structured meeting schedule and central support staff), but the content 
and focus of professional learning and coaching support were not established until well into the 2015-16 
school year due to this leadership transition. As such, this focus was not fully implemented until 2016-17 
(the second year of ESM implementation). 


ESM Expenditures 


In 2017-18, ESM expenditures were $2,490,451 (see Table 1). The vast majority ($1,981,155 or 80%) of 
ESM expenditures was federally funded by Title | dollars. A smaller balance of the total expenditures was 
supported by local funds ($453,484 or 18%) and state funds ($55,813 or 2%). Overall, the ESM 
expenditures were reduced by approximately one million each year of implementation. The 2015-16 
budget (year 1) of $4,500,724 was reduced to $3,499,540 in 2016-17 and to $2,490,451 in 2017-18. 
Reductions in ESM funding could be explained by changes in Title | funding requirements. ESM was 
funded by a Title | “set aside” in 2015-16 (year 1). However, this one-year waver did not apply in years 2 
and 3; thus, ESM schools received similar funding to other Title | schools with similar free or reduced- 
price lunch (FRL) populations. 
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Table 1 
2017-18 (Year 3) Expenditures by Funding Source 


Federal Local State Total 
Central-Based Salary 327,986 39,573 367,558 
Salary/Benefits 451,451 112,753 16,240 580,444 
School-Based Salary 1,242,318 1,242,318 
Supplies and Materials 3,202 3,202 
Training/Salary 287,386 5,242 292,627 
Misc. 2,536 2,536 
Travel 1,767 1,767 
Grand Total $1,981,155 $453,484 $55,813 $2,490,451 


Data Source: WCPSS Budget Department 
Note: All figures have been rounded to the nearest dollar which may have resulted in slightly different totals. 


Furthermore, Title | schools with similar FRL percentages were required to provide coaching at the same 
level (i.e., two coaches per school) and in 2016-17 ESM schools were required to participate in district- 
wide professional development. These changes resulted in a reduction in specific ESM funding above 
and beyond what other Title | schools received. 


Comprehensive School Reform 


In essence, ESM can be considered a customized schoolwide support model, or comprehensive school 
reform (CSR) model. CSR models are school-level initiatives that are intended to improve entire schools 
instead of implementing a series of independent improvement efforts (Borman, Hewes, Overman, & 
Brown, 2003). Often, the goal of implementing these models is to improve instruction at continually 
low-performing schools in an effort to improve educational outcomes for at-risk students. Research on 
CSR has found that the success of these reform efforts rests on the quality of implementation (Datnow, 
Borman, & Stringfield, 2000; Stringfield et al., 1997); internally developed CSR models tend to be 
implemented with less fidelity than those externally developed (Bodilly, 1996, 1998; Nunnery, 1998). 
Effective CSR models emphasize professional development and support during initial implementation, 
and stakeholder buy-in is critical to the success of these initiatives (Borman et al., 2000; Datnow & 
Stringfield, 2000). 


Leadership / Mentoring 


One major component of the WCPSS ESM initiative is leadership mentoring provided by the area 
superintendent and the ESM instructional team. North Carolina’s Standards for School Executives focus 
on seven main facets of principal leadership: Strategic (including vision and improvement plans), 
Instructional, Cultural, Human Resource (including evaluating, developing, and hiring teachers), 
Managerial (including budget management and interpersonal relationships), External Development 
(including family/community outreach and compliance with state and district mandates), and Micro- 
Political (including creating an environment where school staff can take on leadership opportunities and 
safely deliver feedback) (NCDPI, 2011). By having fewer schools within the ESM area, the ESM central- 
based support team (including the ESM Area Superintendent) are able to have regular (monthly or more 
frequent) structured meetings with school administrators with the goal of removing administrative 
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obstacles (e.g., providing hiring priority) and strengthening principals in areas consistent with the NC 
Standards for School Executives. Moreover, research literature on principal leadership generally favors 
principals who espouse a school vision, create goals based on that vision, and support teachers and staff 
in realizing these goals (Marshall, 2015; Rutherford, 1985; Saban & Wolfe, 2009). As such, the structured 
meeting schedule and the leadership/mentoring components of ESM are designed to enable adequate 
time and focused collaboration with school administration around these key areas. 


Principal mentorship and coaching represent a key method of helping principals develop leadership skills 
(Bush & Chew, 1999). Conceptualizing a successful mentoring relationship is difficult given that 
characteristics of the school environment, the mentee, and the mentor all contribute to a relationship’s 
success (Dominguez & Hager, 2013; Parylo, Zepeda, & Bengston, 2012). Successful mentorship 
relationships are based on mutual trust, shared visions and goals, and effective communication of 
strengths and weaknesses (Armstrong, Allinson, & Hayes, 2002; Bloom, Castagna, & Warren, 2003; 
Dominguez & Hager, 2013; Parylo, Zepeda, & Bengston, 2012). One model of mentorship is the Coaching 
Leaders to Attain Student Success model (CLASS), as described by Bloom, Castagna, and Warren (2003). 
According to this model, successful mentoring relationships: 1) offer a fresh perspective of 
circumstances and possibilities, 2) are based upon trust and permission, 3) utilize instructional and 
facilitative strategies based on experience and reflection, 4) focus on student success, and 5) develop 
goals based on professional standards (Bloom et al., 2003). While the CLASS model is generally used to 
train principal coaches who have no authoritative power over principals, it may also have benefits in the 
informal mentorship between the area superintendent and ESM principals irrespective of the 
supervisor/subordinate component to the relationship that also exists in this setting. 
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Program Goals 


The Elementary Support Model (ESM) had several goals as shown in Figure 1. The primary goal was for 
ESM to increase student learning by strengthening teacher quality and school leadership. 


Effort: Elementary Support Model 
Need: Through a close review of data and a formal presentation process involving both school staff and 
Central Services staff, the district has identified 12 elementary schools that have been consistently low 


Figure 1 
Pathway of Change 


performing as measured by state assessments and additional data points. The selected schools have a higher 
number of students with chronic absenteeism, homelessness, and critical needs that negatively impact student 
achievement. Additionally, these schools have approximately 190 new beginning teachers (BT1s—65; BT2s— 
58; and BT3s—67) who require four observations each on a yearly basis. 


les 


Strate 


e 12 ESM schools 
are grouped and 
supervised by one 
person and 
supported by a 
cross-functional 
team 


The model 
addresses staffing, 
professional 
development, 
governance, 
resources, 
calendars, and 
schedules 


c| ® Teachers receive an 


io 


lementat 


im 


additional 10 days of 
PD designed to 
strengthen core 
instruction and 
improve teacher 
quality 


Cross-functional 
instructional team 
supports schools’ 
needs 


Full-time social 
worker and IRT at 
each school 
supports students 
with highest needs 


Improved calendar 
(i.e., one calendar 
for all ESM schools) 
and master 
schedules adopted 
at all ESM schools 


-Term Outcomes 


J | Short 


e PD sessions are 
well-attended, 
increase 
knowledge, and are 
applicable to staffs' 
work 


Teachers are 
implementing PD 
with fidelity 


-Term Outcomes 


Support from ESM 
team, principal, 
and coaches is 
rated as “helpful” 


School social 
worker identifies 
and supports 
students to meet 
critical needs 


Improve student 
performance 


Improve teacher 
quality and 
retention 


Strong leadership 


Improve attendance 
and engagement 
among students 


Achievement gaps 
are reduced 


EOG proficiency 
increases 


EVAAS growth 
increases 


Middle school 
student outcomes 
improve 


ESM 2014-15 to 2017-18 April 2019 


AY Cod etayers 


The 12 ESM elementary schools were selected based on a pre-ESM ranking based on multiple factors. In 
2014-15, the 12 elementary schools with the lowest composite Multiple Factors Index (MFI) scores were 
designated as ESM schools based on 2013-14 data. The factors within the MFI included achievement 
(e.g., percentage of students scoring at “Benchmark” level on the DIBELS composite, and percentage of 
students scoring proficient on end-of-grade exams), teacher characteristics (e.g., teacher turnover rate), 
demographics (e.g., percentage of students with disabilities), school climate (e.g., short-term 
suspensions), and school leadership (e.g., teacher survey results). Utilizing a variety of data sources (see 
Table 2), we used a quasi-experimental design to examine ESM’s intermediate and long-term outcomes. 


Data Sources 
Table 2 
Data Sources for the Study 


Data Source 2014-15 2015-16 2016-17 2017-18 
(baseline) 


Principal Focus Groups 
(2 focus groups with 6 principals in each) 
ESM Central Services Administrative Staff Focus Group 


Professional Development SRN Data 
WCPSS Annual Teacher Survey 
Walkthrough or “look for” data collected from ESM schools 


x 


Teacher absences tables provided by Human Resources 


Teacher counts provided by Human Resources 


Educator Effectiveness file from NCDPI 
Student Data (Demographics, EOG, and DIBELS) 
WCPSS Annual Student Engagement Survey 


SAINI NIA SN 

5) ul See SN 

STAIN SENSIS SON 
SINISE 


Note: Blanks indicate data not collected for that year. 
NCDPI = North Carolina Department of Public Instruction 


Study Design 


This evaluation is based on a quasi-experimental design where each ESM school was matched to a 
comparison school. Twelve comparison elementary schools were selected based on a matching 
procedure that accounted for school and student-level characteristics. The study’s relatively small 
sample size of 24 schools (12 ESM and 12 comparison schools) meant that the study had limited power 
to detect significant differences. In other words, the impact of ESM would need to be large for 
significant effects to be found. Another limitation of the study design was that the absolute lowest 12 
schools (based on the multiple factors) were designated ESM which made finding comparison schools 
problematic. To help strengthen the similar comparisons between ESM (ranked the 12 lowest in 2013- 
14) and comparison schools (ranked above ESM in 2013-14) schools were matched both on school and 
student-level characteristics. 
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Multilevel Analyses 


In addition to descriptive analyses conducted throughout, we conducted a series of multilevel analyses 
to control for possible effects of prior student achievement, student demographic characteristics, and 
school effects. Such statistical control allowed for a more precise measurement of outcomes and 
improved the ability to attribute any observed results to the implementation of ESM. To explore 
whether ESM had any differential effects on subgroups of students, we included cross-level interactions 
between the school-level predictor (ESM vs. comparison schools) and student-level predictors, such as 
Limited English Proficient (LEP) status and special education (SWD) status in our model. 


Table 3 
Nature of the Data Provided and Valid Uses 


Research Design Conclusions that Can be Drawn 


QO Experimental We can conclude that the program or policy caused changes in 
outcomes because the research design used random assignment. 

I Quasi-Experimental We can reasonably conclude that the program or policy caused 
changes in outcomes because an appropriate comparison strategy 
was used. 

VI Descriptive These designs provide outcome data for the program or policy, but 


differences cannot be attributed directly to it due to lack of a 


M Quantitative comparative control group. 
QO Qualitative 


Sources: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014) 


Implementation 


Program Implementation: Was ESM implemented as planned? 


Implementation goals were examined in 2015-16 (year 1) and 2016-17 (year 
2) to establish fidelity of ESM implementation. As of the 2016-17 school year, 
four of the six implementation goals were fully met, and two goals was 
mostly met. 


Prior to examining ESM outcomes, it is imperative that the strength of program implementation be 
determined. Thus, the extent to which ESM was implemented with fidelity in 2015-16 (year 1) and 2016- 
17 (year 2) was evaluated and is reflected in Table 4: 


e Structured ESM meetings were scheduled as planned, and the tiering support to ESM schools was 
viewed as more effective in meeting individual school needs. 

e The regular support and guidance received by principals from the Central Services ESM Team 
members was appreciated. 
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e Principals reported that in 2015-16 a vendor developed walkthrough tool was adopted but never 
made available to schools; thus, a Google form for classroom observations was created by ESM 
Team members and Human Resources staff specifically for ESM schools. While principals utilized the 
Google form in 2016-17, they reported that the tool was too general to meet their school’s needs. 
Therefore, in 2015-16 and for part of 2016-17 principals used their own tools and methods to 
conduct the majority of formal and informal observations. In 2017-18, a districtwide ‘look for’ tool 
was adopted. However, these data were used formatively and thus not captured centrally. 

e While the principals found having two full-time coaches beneficial, they suggested that the role of 
the ESM coach should be more flexible and responsive to school needs. In 2016-17, each school was 
assigned two full-time coaches (an instructional and a content area coach) in an effort to respond to 


the reported gaps in service caused by sharing coaches in 2015-16. 


e Master schedules were adopted; however, calendar changes were not approved by WCPSS’ Board. 
e Each school had a full-time social worker and instructional resource teacher to support students 


with the highest needs. 


Table 4 
Implementation Status Ratings 
Implementation Goals 2015-16 2016-17 
Governance 
1. Structured ESM meetings scheduled as planned and included a 
. : tote ke | ei 
tiered system of support 
2. Principal mentoring/coaching provided by ESM Area Superintendent N/A Kk *& 
Professional Development and Coaching 
3. Walkthroughs or “look fors” are conducted regularly at all 12 schools | yi sy * ks 
4. Two full-time school-based coaches for each ESM school N/A > a. oa 
Calendars and Schedules 
5. One ESM calendar and master schedules adopted at all ESM schools | yheyiew'> | oie tie tr 
Staffing 
6. Full-time social worker and IRT at each school supports students with 
i oe ab & as 
highest needs 
Goal Status: fully met mostly met partially met not met 
KKK kK Kk Wie 
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Has ESM met its short-term goals? 


Three of the five ESM short-term goals were fully met (goals related 
to Professional Development and coaching), one was mostly met 
(goals related to the use of walkthrough or “look for” data) and one 
goal—teachers implementing targeted best practices with fidelity— 
could not be determined due to a lack of baseline data. 


e Within the majority (54%) of ELA lessons observed there was evidence of small group instruction 
and/or guided reading groups, which are considered best practices and emphasized within the 
ESM professional development. However, the annual increase in teachers implementing 
targeted best practices with fidelity could not be determined due to a lack of two years of 
available data (i.e., no baseline data in 2015-16 and no centrally collected data in 2017-18). 

e The vast majority (>80%) of teachers reported positively on the effectiveness of ESM 
professional development in meeting classroom needs in 2016-17. 

e In 2017-18, 47% of teachers in ESM schools reported having received coaching at least monthly 
or with greater frequency, compared to 33% of teachers in comparison schools and 25% in non- 
ESM schools (results were similar in prior years). 

e In 2017-18, a higher percentage of teachers at ESM schools (92%) reported favorably that 
coaching was applicable to their work than did teachers at comparison and non-ESM schools 


(86%). 
Table 5 
Short-term Goals Status Ratings 
Short-term Goals 2015-16 | 2016-17 | 2017-18 
Governance 


1. Walkthrough or “look for” data are used to inform instruction [sexe s’e | ieee N/A** 


BA IN 


2. Annual increase in teachers implementing targeted best N/A 
practices with fidelity 
Professional Development 


Wey | N/A** 


3. Staff report that training sessions increased their knowledge 
and were applicable to their work 


Kk kk *& N/A* 


Coaching 
4. Teachers report that coaching occurred regularly KKK kk Kk KK *& 
5. Teachers report coaching was applicable to their work tote: Keke tek 


Notes: * This question was not asked in 2017-18. 
** A common ”Look-For” tool has been adopted by the district and will be used to inform instruction; however, use of 
the tool was in the initial stages of implementation in 2017-18 and data were not collected centrally. 

Goal Status: fully met mostly met partially met not met 


> @ ano One Kk exeXe 
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Has ESM met its intermediate and long-term goals? 


Three of the 12 intermediate goals were fully met in 2017-18 (goals 
related to leadership support, a reduction in the percentage of 
beginning teachers, and schools meeting expected growth). Six goals 
were partially met—percentage of highly effective teachers, teacher 
absenteeism, student-teacher relationships, 6" grade student 
achievement, and reduction in achievement gaps. Three goals were 
not met. 


School Leadership Support: 


The percentage of ESM teachers reporting that they feel supported by leadership increased 
seven percentage points from 2014-15 (prior to ESM) to 2017-18 (year 3). 


Teacher Outcomes: 


The average days of leave taken by ESM teachers increased by half a day from 2014-15 to 2017- 
18 while it increased one day at comparison schools. 

There was a seven percentage point reduction in the proportion of beginning teachers from 
2014-15 (prior to ESM) to 2017-18, compared to a reduction of four percentage points at 
comparison schools. 

One of the five North Carolina Educator Evaluation System (NCEES) standards, Standard I- 
Teachers Demonstrate Leadership, had a four-percentage point increase in teachers rated as 
Accomplished or Distinguished from 2014-15 and 2017-18 while the remaining four standards 
remained approximately the same. 


Student Outcomes: 


The overall goal of >9% of students with chronic absenteeism and an annual 2 percentage point 
decrease was partially met. The ESM average chronic absence trend followed a similar pattern 
to the comparison and non-ESM schools’ trends. In 2017-18, six out of the 12 ESM schools 
experienced a reduction in chronic student absenteeism from prior to ESM. 

The goal of 80% agreement with teacher-student relationship statements was reached on only 
four out of nine teacher-student relationship survey questions each year examined. 

The percentage of students meeting DIBELS benchmarks in grades K-5 remained stable at ESM 
schools; thus, the goal of a three percentage-point increase of students at benchmark was not 
met. 

The goal of 57% of students proficient (5 percentage point increase) on the EOG performance 
composite was not met. 

Eleven out of 12 (92%) ESM schools met or exceeded growth (as measured by EVAAS); thus, the 
goal of 83% was met. 

No schools exceeded growth; therefore, the goal of 25% of schools (three out of 12) exceeding 
expected growth was not met. 
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e The goal of a two percentage point reduction in achievement gaps annually was partially met. 
While there was a total of two percentage point reductions from 2014-15 (prior to ESM) to 
2017-18, the annual two percentage point decrease was not reached. 

e The goal of improved grade 6 outcomes for ESM students once they get to middle school was 
partially met. While the percentage of 6" grade students proficient on their reading EOG 
increased six percentage points from 2014-15 to 2017-18, math proficiency decreased one 
percentage point. 


ESM did not show any statistically significant effects across student subgroups. 


Table 6 


Intermediate and Long-term Goals Status Ratings 


Intermediate and Long-term Goals 2015-16 2016-17 2017-18 
School Leadership Support 
1. Increase in teachers reporting they feel supported b 
ee ue u N/A KKK | kk *& 
school leadership compared to prior years 
Teacher Outcomes 
2. Decrease in teacher absenteeism compared to prior hcalicatin rae ae 
as WY Kk | Kei 
3. Reduction in the percentage of BTs compared to tr 
* 
prior years * KKK | kk * 
4. Increase in the percentage of highly effective a oa 
: . eu N/A Kk Keck 
teachers compared to prior years 
Student Outcomes 
5. 9% of students with chronic absenteeism (2 a om car a 
: N/A Tf Pak Ra te vy 
percentage point decrease) 
6. 80% or greater agreement on all Teacher-Student oy —_ 
ns z ‘ e : N/A Ke iexk Ket 
Relationship survey items 
7. 64% of students at benchmark (3 percentage point ‘is iliac lcslies whoraheahy 
increase) on DIBELS composite / 
8. 57% of students proficient (5 percentage point iar ees Pa ae 
increase) on EOG performance composite 
9. 83% of schools (10 out of 12) meet or exceed ro 
expected growth (as measured by EVAAS data) wk | KKK 
10. 25% of schools (3 out of 12) exceed expected growth | eyhe oe | yess’: | siesies’> 
11. 2 percentage point reduction in achievement gaps ra iy 
: Bee BAP N/A > kek Kix 
annually 
12. Improved grade 6 achievement for ESM students N/A N/A tees 
Goal Status: fully met mostly met partially met not met 
KKK kK Kk yess 
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School Leadership Support 


Has the percentage of teachers who report feeling supported by school leadership increased from 2014- 
15 to 2017-18 when compared to teachers in comparison schools? 


The percentage of teachers reporting consistent support from school leadership increased steadily at 
ESM schools from 2014-15 to 2017-18 (66% to 73%), while at comparison schools the year-to-year 
difference fluctuated. The percentage point increase in teacher agreement increased seven percentage 
points at ESM schools versus a four percentage point decrease among teachers at comparison and non- 
ESM schools (this was a statistically significant difference). The percentage of teacher agreement 
remained higher at comparison schools and non-ESM schools than at ESM schools, however, the gap 
closed by 11 percentage points for both groups of schools. 


Figure 2 
Percentage of Teachers Reporting Consistent Support by Leadership at 
ESM, Comparison, and non-ESM Elementary Schools, 
2014-15 to 2017-18 
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Teacher Absence Rate 


Has the teacher absence rate decreased from 2014-15 (prior to ESM implementation) when compared to 
teachers at comparison schools? 


The overall teacher absence rate increased for both the ESM schools and the comparison schools over the 
four-year period. However, the increase for ESM was only by half a day on average, in contrast to the 
increase for the comparison schools that was 1.3 days on average. In 2017-18, the average teacher 
absence rate was lower at the ESM schools than at the comparison schools (9.2 vs 10.2). 


Figure 3 
Teacher Absence Rate 2014-15 to 2017-18 
mESM Comparison & Non-ESM 
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Note: Teacher attendance is the combination of sick leave, personal leave, donated leave, 
and bonus leave. Annual leave was not included. 
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Reduction of Beginning Teachers 


Has the proportion of beginning teachers decreased in ESM schools compared to prior years and 
teachers at comparison schools? 


The overall proportion of beginning teachers (teachers with three or fewer years of experience) at ESM 
schools and at the comparison schools decreased from 2014-15 (prior to ESM) to 2017-18. The 
proportion of BTs at ESM schools has remained higher than at the comparison schools over the four-year 
period, however the ESM schools had a faster rate of decrease (6.7 percentage points for ESM schools 
and 3.8 percentage points for the comparison schools). 


Figure 4 
Percentage of Beginning Teachers at ESM, Comparison, and Non-ESM Schools 
2014-15 to 2017-18 
50% 5 mMESM Comparison & Non-ESM 
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Highly Effective Teachers 


Has there been an increase in the percentage of highly effective teachers compared to prior years (with 
2014-15 being the baseline year)? 


Only one of the five NCEES standards, Standard | (Teachers Demonstrate Leadership), had a four- 
percentage point increase in teachers rated as Accomplished or Distinguished from 2014-15 and 2017- 
18, while the remaining four standards remained approximately the same. 


Figure 5 
Percentage of ESM Teachers NCEES Rating by Standard, 2014-15 to 2017-18 
65% 
60% 60% 
60% 59% 
58% 
55% 
50% 


49% 9, 
48% Te 49% 
45% 44% 


40% 41% 


35% 
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—Standard1 = Standard2 == Standard3 ==Standard4 Standard 5 


Note: Percentages shown are for teachers rated as Accomplished or Distinguished. 
Standard |: Teachers Demonstrate Leadership 
Standard II: Teachers Establish a Respectful Environment for Diverse Population of Students 
Standard III: Teachers Know the Content They Teach 
Standard IV: Teachers Facilitate Learning for Their Students 
Standard V: Teachers Reflect on Their Practice 
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Student Absences 


Has chronic student absenteeism decreased from 2014-15 (baseline prior to ESM implementation) when 
compared to teachers at comparison schools? 


Among ESM schools across the observed years, chronic absenteeism ranged from a low of 5.6% (Fox 
Road Elementary, 2014-15) to a high of 18.7% (Walnut Creek Elementary, 2017-18). The overall goal of 
9% of students with chronic absenteeism and an annual 2 percentage point decrease was partially met. 
In 2017-18, if the outlying high of 18.7% is removed, the average chronic absenteeism percentage drops 
from 10.7% to 9.9%, which approaches achievement of the overall goal of 9%. Moreover, in 2012-13 
across ESM schools, chronic absenteeism was widespread with a range of 11.6 percentage points. By 
2017-18, the spread decreased to a range of 6.6 percentage points (excluding the outlying school). 


Figure 6 
Percentage of Students with Chronic Absenteeism at ESM, Comparison, and Non-ESM Schools, 


2012-13 to 2017-18 
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The average chronic absence rate for all ESM schools (dashed line) shows that on average, ESM schools 
reduced chronic absenteeism up to 2015-16, but there was an uptick in 2016-17 which held steady in 
2017-18. 


Figure 7 
ESM Absence Rates by ESM School Compared to All ESM Schools’ Average, 2012-13 to 2017-18 
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Student Engagement 
Has student engagement increased from 2014-15 (the baseline year)? 


In 2017-18, in response to the teacher-student relationships questions, students at ESM elementary 
schools either positively agreed comparably, or had higher rates of positive agreement than students at 
comparison schools on about half of the items considered. For ESM elementary schools, the goal of 80% 
agreement was reached on only four out of nine teacher-student relationships items; thus, the goal was 
partially met. While the percentage of students agreeing increased between 2016-17 and 2017-18 for 
seven out of nine items, ESM students showed a decrease in engagement from prior to ESM (2014-15) 
to 2017-18 on eight of the nine items (this pattern also took place in comparison schools). 
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Figure 8 
Percentage of Student Agreement on Teacher-Student Relationship Items: ESM, Comparison, and Non-ESM Schools, 2014-15 to 2017-18 
ESM Comparison Non-ESM 
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Note: There has been a decline in average student response rates within WCPSS elementary schools: 90% in 2015-16, 91.1% in 2016-17, and 83% in 2017-18. 
East Garner Elementary and Lynn Road Elementary had a 0% student response rate in 2017-18. Therefore, 2017-18 ESM school percentages do not 
include student data from these two schools. Green Elementary (a comparison school) had only one student response. Therefore, 2017-18 comparison 
school percentages do not include complete student data from this school. 
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Student Performance 
Has student performance increased from 2014-15 (the baseline year) to 2017-18? 


The goal of a three-percentage point increase of students meeting DIBELS benchmarks was not met. The 
percentage of students meeting DIBELS benchmarks remained stable at ESM schools while it decreased 
slightly (four percentage points) among comparison and non-ESM schools from 2014-15 (prior to ESM) 
to 2017-18 (see Figures 10 and 11). Similar results were found among students scoring “Well Below” 
based on DIBELS Benchmarks, with the percentage of ESM students “Well Below” remaining consistent, 
but increased four-percentage points for students at comparison schools, and three-percentage points 
for students at non-ESM schools (see Figure 9). Additionally, changes from prior to ESM to 2017-18 
favored ESM schools over comparison and non-ESM schools for each subgroup considered; however, 
these differences were not significant. 


The goal of 57% of students proficient (five percentage point increase) on EOG performance composite 
was not met (see Figures 12 and 13). In 2017-18, 11 out of 12 (92%) ESM schools met or exceeded 
growth (as measured by EVAAS); thus, the goal of 83% was met (see Table 7). However, no schools 
exceeded growth; therefore, the goal of 25% of schools (three out of 12) exceeding expected growth 
was not met. Additionally, while the goal of a two percentage point reduction in achievement gaps 
annually was partially met, there was a two percentage point reduction from 2014-15 (prior to ESM) to 
2017-18. The goal to improve grade 6 outcomes for ESM students once they get to middle school was 
only partially met, with a nonsignificant increase in reading percent proficient and no change in math 
from baseline to 2017-18. From 2014-15 to 2017-18 there was a six-percentage points increase in 
reading proficiency among grade 6 students who had attended ESM schools. While math proficiency 
increased in 2015-16, it decreased in 2016-17 and 2017-18, resulting in a one percentage point 
reduction across years. 


Figure 9 
Percent of Students Scoring “Well Below” based on DIBELS Benchmarks for 
ESM, Comparison, and Non-ESM Schools from 2014-15 to 2017-18 
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Figure 10 


April 2019 


Percent Meeting DIBELS Benchmarks for ESM, Comparison, and Non-ESM Schools from 2014-15 to 


2017-18 by Student Demographics 
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Note: EDS — Economically disadvantaged students; SWD — Students with disabilities; LEP — Limited English Proficient 


Figure 11 


Percent Meeting DIBELS Benchmarks for ESM, Comparison, and Non-ESM Schools from 2014-15 to 


2017-18 by Student Race/Ethnicity 
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Figure 12 
Percent of EOG Test Scores Proficiency (i.e., Levels 3, 4, & 5) for ESM, Comparison, and Non-ESM 
Schools from 2014-15 to 2017-18 by Student Demographics 
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Notes: 1. Increases in limited English proficient (LEP) students’ performance may reflect an LEP classification change (LEP to EL). 
2. EDS — Economically disadvantaged students; SWD — Students with disabilities 


Figure 13 
Percent of EOG Test Scores Proficiency (i.e., Levels 3, 4, & 5) for ESM, Comparison, and Non-ESM 
Schools from 2014-15 to 2017-18 by Student Race/Ethnicity 
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Table 7 
EVAAS Status: ESM Schools 


Years (out of 4) 


SCHOOL NAME 2014-15 2015-16 2016-17 2017-18 Met or Exceeded 
Barwell Road Elementary Met Exceeded Met Met 4 
Brentwood Elementary Not Met Met Met Met 3 
Bugg Elementary Not Met Not Met Not Met Met 1 
Creech Road Elementary Met Exceeded Not Met Not Met 2 
East Garner Elementary Met Met Met Met 4 
Fox Road Elementary Met Exceeded Met Met 4 
Hodge Road Elementary Not Met Exceeded Exceeded Met 3 
Lincoln Heights Elementary Not Met Met Met Met 3 
Lynn Road Elementary Exceeded Met Met Met 4 
Smith Elementary Met Not Met Met Met 3 
Walnut Creek Elementary Met Not Met Not Met Met 2 
Wilburn Elementary Met Exceeded Not Met Met 3 
Total (Met or Exceeded) 8 (67%) 9 (75%) 8 (67%) 11 (92%) 

Barwell Road Elementary Met Met Met Not Met 3 
Brentwood Elementary Not Met Met Met Met 3 
Bugg Elementary Not Met Not Met Not Met Met 1 
Creech Road Elementary Met Exceeded Not Met Not Met 2 
East Garner Elementary Met Met Not Met Not Met 2 
Fox Road Elementary Met Exceeded Met Exceeded 4 
Hodge Road Elementary Met Met Met Met 4 
Lincoln Heights Elementary Not Met Met Met Met 3 
Lynn Road Elementary Met Met Not Met Met 3 
Smith Elementary Met Not Met Met Met 3 
Walnut Creek Elementary Met Met Met Met 4 
Wilburn Elementary Met Met Met Met 4 


Total (Met or Exceeded) 9 (75%) 10 (83%) 8 (67%) 9 (75%) 


Barwell Road Elementary Met Exceeded Met Met 4 
Brentwood Elementary Met Met Met Met 4 
Bugg Elementary Met Met Met Met 4 
Creech Road Elementary Exceeded Met Not Met Not Met 2 
East Garner Elementary Met Exceeded Exceeded Met 4 
Fox Road Elementary Met Met Met Met 4 
Hodge Road Elementary Not Met Exceeded Exceeded Met 3 
Lincoln Heights Elementary Met Met Exceeded Met 4 
Lynn Road Elementary Met Met Exceeded Met 4 
Smith Elementary Met Not Met Met Met 3 
Walnut Creek Elementary Met Met Not Met Met 3 
Wilburn Elementary Met Exceeded Not Met Not Met 2 
Total (Met or Exceeded) 11 (92%) 11 (92%) 9 (75%) 10 (83%) 
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Figure 14 
Percent of 6" Grade EOG Test Scores Proficiency (i.e., Levels 3, 4, & 5) for ESM Students 
from 2014-15 to 2017-18 by Subject 
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Graph Interpretation Note: In 2013-14, the 12 elementary schools with the lowest composite Multiple Factors 
Index (MFI) scores were designated as ESM schools. These schools are displayed in order from 1-12 at the date 
marked "2014" (i.e., the 2013-14 school year). The four subsequent years show how this initial ranking 

changed based on updated MFI calculations. For example, by the 2017-18 school year ("2018") four of the 12 ESM 
schools had MFI scores that placed them above the original bottom 12 slots. 
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Table 8 
EVAAS Status: ESM Schools 


Multiple Factors Index Rank from Bottom 


School 2014 2015 2016 2017 
Bugg ES 9 6 1 1 
East Garner ES 12 7 5 2 
Barwell Rd. ES 8 5 2 3 
Hodge Road ES 1 1 3 4 
Smith ES 7 10 7 5 
Creech Road ES 3 4 6 6 
Walnut Creek ES 2 3 4 8 
Wilburn ES 5 2 10 10 
Lincoln Heights ES 10 14 13 11 
Brentwood ES 6 13 16 17 
Lynn Road ES 11 17 27 27 
Fox Road ES 4 9 18 28 


Note: Schools are ranked from lowest to highest; thus, a number | indicates the school that ranked the lowest based 
on the multiple factors index and had the greatest need for support. 


Discussion 


Overall ESM implementation was strong (four out of six implementation goals were fully met in 2016-17 
and two were mostly met). Short-term goals were also either mostly met or fully met with one 
exception: “teachers implementing targeted best practices with fidelity” (which could not be 
determined due to lack of centrally collected walkthrough or “look for” data). Results related to ESM 
intermediate and long-term goals were mixed. While three out of 12 goals were met, six were partially 
met, and three were not met. It should be noted that implementation science suggests it takes three to 
five years for full implementation of a significant change initiative like ESM (Fixsen, Naoom, Blasé, 
Friedman, & Wallace, 2005). Thus, it is not surprising that only a quarter of intermediate and long-term 
goals were fully met in year three of ESM implementation. 


Three areas of strength were principal support to teachers, reduction of beginning teachers, and 
increases in the number of ESM schools meeting EVAAS growth. In order to realize gains in student 
proficiency and reduce achievement gaps it is necessary to strengthen support to teachers and improve 
teacher outcomes; thus, it is encouraging that ESM was able to reduce the percentage of beginning 
teachers and increase the percentage of teachers reporting they consistently felt supported by their 
principal. However, other ESM goals related to teacher attendance (partially met) and NCEES standards 
(partially met) still need to be fully realized. Additionally, goals related to teacher-student relationships 
(partially met) and student attendance (partially met) would need to be fully met in order to improve 
the chances that student achievement goals are met. The fact that the EVAAS goal to increase the 
percentage of schools meeting growth was met (11 of 12 ESM schools met growth) is an encouraging 
finding given growth is necessary if additional proficiency goals related to DIBELS composite (not met) 
and End-of-Grade (not met) are to be met. None of the ESM schools exceeded growth in 2017-18, which 
is problematic given accelerated growth is required for ESM schools to no longer be among the district’s 
lowest performing schools. Furthermore, while there was a slight reduction in achievement gaps 
between student subgroups (two percentage points from baseline to 2017-18), the goal to reduce gaps 
by two percentage points annually was only partially met. 
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Recommendations 


Based on this study’s findings generated from analyses of a variety of data sources (e.g., focus groups, 
survey data, and student achievement outcomes), we have the following recommendations: 


e More consistent school to school data collection efforts. Clarify expectations regarding the 
completion of student surveys. Centralize the collection of walkthrough or “look for” data. 

e Utilize an observational tool to inform instruction. School staff should use a ‘look for’ tool in order 
to inform teacher planning and monitor the success of classroom implementation of professional 
learning and the application of learning shared via coaching support. Use of the ‘Look-for’ should 
provide school administrators and other school staff formative information on classroom practices 
going forward. 

e Unpack principal concerns regarding ESM Coaches’ roles. Based on principal feedback, the role of 
the two full-time ESM coaches should be more flexible and responsive to school needs. ESM support 
teams focus-group feedback indicated that beginning in 2017-18, each individual school’s needs 
would guide the coach’s role at that school. 

e Clarification regarding accessing support from coordinating teachers. ESM principals requested 
improved communication for accessing support from coordinating teachers and clarification on the 
process for receiving that support. 

e Continue to align ESM goals with MTSS Framework. Given that 2017-18 represents the third year of 
ESM implementation, and according to implementation science it takes three to five years to realize 
full implementation, ESM staff should revise (annually) and continue use of the ESM logic model as a 
tool for tracking progress toward reaching ESM goals. Furthermore, since the MTSS Framework is 
the broader umbrella under which the ESM initiative fits, staff should ensure ESM goals continue to 
align with the MTSS Framework. 

e Use successful ESM strategies and ESM school exemplars to inform the district’s Collaborative 
System of School Support. ESM’s structured support and leadership/mentoring strategy have 
shown success in reducing the number of beginning teachers, increasing the percentage of teachers 
feeling consistently supported by leadership, and increasing school’s meeting EVAAS growth. 


27 


ESM 2014-15 to 2017-18 April 2019 


References 


Armstrong, S. J., Allinson, C. W., & Hayes, J. (2002). Formal mentoring systems: An examination of the effects of 
mentor/protégé cognitive styles on the mentoring process. Journal of Management Studies, 39(8), 1111-1137. 


Bloom, G., Castagna, C., & Warren, B. (2003). More than mentors: Principal coaching. Leadership, 32(5), 20-23. 

Bodilly, S. J. (1996). Lessons from New American Schools Development Corporation’s demonstration phase. Santa 
Monica, CA: RAND. 

Bodilly, S.J. (1998). Lessons from New American Schools’ scale-up phase: Prospects for bringing designs to multiple 
schools. Santa Monica, CA: RAND. 

Borman, G. D., Hewes, G.M., Overman, L.T., & Brown, S. (2003). Comprehensive school reform and achievement: A 
meta-analysis. American Educational Research Association, 73, 125-230. 


Borman, G. D., Rabucha, L., Datnow, A., Alberg, M., Maclver, M., & Stringfield, S. (2000). Four models of school 
improvement: Successes and challenges in reforming low-performing, high poverty Title | schools. CRESPAR Report 
No. 48. Baltimore: Johns Hopkins University, Center for Research on the Education of Students Placed at Risk. 

Bush, T., & Chew, J. (1999). Developing human capital: training and mentoring for principals. Compare: A Journal of 
Comparative and International Education, 29(1), 41-52. 

Datnow, A., Borman, G., & Stringfield, S. (2000). School reform through a highly specified curriculum: A study of the 
implementation and effects of the Core Knowledge Sequence. Elementary School Journal, 101, 167-192. 

Datnow, A., & Stringfield, S. (2000). Working together for reliable school reform. Journal of Education for Students 
Placed At Risk, 5, 183-204. 

Dominguez, N., & Hager, M. (2013). Mentoring frameworks: synthesis and critique. International Journal of Mentoring 
and Coaching in Education, 2(3), 171-188. 

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of 
the literature. Tampa, FL: University of South Florida, National Implementation Research Network. 

List, J. A., Sadoff, S. & Wagner, M. (2011). So you want to run an experiment, now what? Some simple rules of thumb for 
optimal experimental design. Experimental Economics, 14(4), 439-457. 

Marshall, C. (2015). "School Leadership and Politics". In English, F., The Sage Guide to Education Leadership and 
Management. Thousand Oaks, CA: Sage 

North Carolina Department of Public Instruction (2011). North Carolina Standards for School Executives. 
http://www.ncpublicschools.org/docs/effectiveness-model/ncees/standards/princ-asst-princ-standards. pdf 

Nunnery, J. (1998) Reform ideology and the locus of development problem in educational restructuring: Enduring 
lessons from studies of educational innovation. Education and Urban Society, 30, 277-295. 

Parylo, O., Zepeda, S. J., & Bengtson, E. (2012). The different faces of principal mentorship. International Journal of 
Mentoring and Coaching in Education, 1(2), 120-135. 

Portin, B., Schneider, P., DeArmond, M., & Gundlach, L. (2003). Making sense of leading schools: A study of the school 
principalship. Seattle, Washington: Center on Reinventing Public Education, Daniel J. Evans School of Public Affairs, 
University of Washington 

Rutherford, W. L. (1985). School principals as effective leaders. Phi Delta Kappan, 67(1), 31-34. 

Saban, J., & Wolfe, S. (2009). Mentoring principals around leadership practices. Catalyst for Change, 36(1). 


Stringfield, S., Millsap, M., Yoder, N., Schaffer, E., Nesselrodt, P., Gamse, B., et al. (1997). Special strategies studies final 
report. Washington, DC: U.S. Department of Education. 
What Works Clearinghouse. (2014). WWC procedures and standards handbook, version 3.0. Washington DC: US 
Department of Education. 
28 


